A method to remove Scale Invariant Feature Transform (SIFT) mismatches based on Canonical Correlation Analysis (CCA) was presented to improve the quality of feature matching when the feature points locate in some similar structures of one image. At first, SIFT matching algorithm was used to get the initial matching pairs. Then, a line was fitted based on the linear relation between the canonical correlation components. The mismatches were removed by thresholding the distances from the points to the line. Furthermore, the influence of each remained match on the collineartiy degree was analyzed to indicate false matches. The experimental results show that the proposed algorithm can remove mismatches efficiently and keep more correct matches.